IN PROGRESS: Deterministic builds #127

alukach · 2024-12-16T22:37:00Z

Experiment with increasing CICD pipeline deployments by avoiding re-deploying services that haven't changed.

Before Change

First Run:

https://github.com/SkyTruth/cerulean-cloud/actions/runs/12399957514/attempts/1

Second Run:

https://github.com/SkyTruth/cerulean-cloud/actions/runs/12399957514/attempts/2

After Change

First Run:

https://github.com/SkyTruth/cerulean-cloud/actions/runs/12381896511/attempts/1

Second Run:

https://github.com/SkyTruth/cerulean-cloud/actions/runs/12381896511/attempts/2

github-actions · 2024-12-16T22:41:47Z

🍹 `preview` on cerulean-cloud-images/test

Pulumi report

Previewing update (test):
@ previewing update........

@ previewing update.............
pulumi:pulumi:Stack cerulean-cloud-images-test running
@ previewing update....
gcp:container:Registry cerulean-cloud-images-test-registry
@ previewing update....
docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image Building your image for linux/amd64 architecture.
~ docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image update [diff: ~build]; Building your image for linux/amd64 architecture.
docker:index:Image cerulean-cloud-images-test-cr-offset-tile-image Building your image for linux/amd64 architecture.
~ docker:index:Image cerulean-cloud-images-test-cr-offset-tile-image update [diff: ~build]; Building your image for linux/amd64 architecture.
@ previewing update....
docker:index:Image cerulean-cloud-images-test-cr-tipg-image Building your image for linux/amd64 architecture.
~ docker:index:Image cerulean-cloud-images-test-cr-tipg-image update [diff: ~build]; Building your image for linux/amd64 architecture.
pulumi:pulumi:Stack cerulean-cloud-images-test
Diagnostics:
docker:index:Image (cerulean-cloud-images-test-cr-orchestrator-image):
Building your image for linux/amd64 architecture.
To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

docker:index:Image (cerulean-cloud-images-test-cr-offset-tile-image):
Building your image for linux/amd64 architecture.
To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

docker:index:Image (cerulean-cloud-images-test-cr-tipg-image):
Building your image for linux/amd64 architecture.
To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

Resources:
~ 3 to update
2 unchanged

github-actions · 2024-12-16T22:43:52Z

🍹 `preview` on cerulean-cloud/test

Pulumi report

   Previewing update (test):

@ previewing update....
   pulumi:pulumi:Stack cerulean-cloud-test running 
@ previewing update....
   pulumi:providers:docker cerulean-cloud-images-test-gcr  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-ais  
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cloud-tasks-ais-analysis  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-ais  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais replace [diff: ~detectMd5hash,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais create replacement [diff: ~detectMd5hash,source]
   gcp:sql:User cerulean-cloud-test-database-users  
   gcp:projects:IAMMember cerulean-cloud-test-cf-ais-iam  
@ previewing update....
   gcp:sql:Database cerulean-cloud-test-database  
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-ais update [diff: ~environmentVariables,secretEnvironmentVariables]
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-ais-invoker  
@ previewing update.......
   gcp:serviceaccount:Account cerulean-cloud-test-cr-offset-tile  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-offset  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator  
   gcp:projects:IAMMember cerulean-cloud-test-cr-offset-tile-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-offset-tile-secretmanagerSecretAccessor  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-offset-tile-secret-accessor-binding  
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cr-offset-tiles update [diff: ~metadata,template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-offset  
@ previewing update......
   aws:iam:Role cerulean-cloud-test-lambda-titiler-role  
   aws:iam:Policy cerulean-cloud-test-lambda-titiler-policy  
   aws:apigatewayv2:Api cerulean-cloud-test-lambda-titiler-api  
   aws:s3:Bucket cerulean-cloud-test-titiler-lambda-archive  
   aws:sns:Topic cerulean-cloud-test-lambda-APIAbuseAlert  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-sr  
~  gcp:cloudtasks:Queue cerulean-cloud-test-queue-cr-orchestrator update [diff: ~rateLimits]
   gcp:serviceaccount:Account cerulean-cloud-test-cr-orchestrator  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-sr  
   gcp:serviceaccount:Account cerulean-cloud-test-cr-tipg  
   aws:iam:Role cerulean-cloud-test-lambda-sentinel1-iam  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment2  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-support  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-aemon  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jason  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jona  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr replace [diff: ~detectMd5hash,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr create replacement [diff: ~detectMd5hash,source]
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run replace [diff: ~detectMd5hash,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run create replacement [diff: ~detectMd5hash,source]
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudTasksEnqueuer  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cf-sr-iam  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-secretmanagerSecretAccessor  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-secretmanagerSecretAccessor  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-sentinel1-basic-execution  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-orchestrator-secret-accessor-binding  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-tipg-secret-accessor-binding  
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cr-tipg update [diff: ~metadata]
~  gcp:cloudrun:Service cerulean-cloud-test-cr-orchestrator update [diff: ~metadata,template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-tipg  
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-orchestrator  
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-historical-run update [diff: ~secretEnvironmentVariables]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-sr update [diff: ~secretEnvironmentVariables]
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-historical-run-invoker  
   aws:lambda:Function cerulean-cloud-test-lambda-sentinel1-sub  
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-sr-invoker  
   aws:sns:TopicSubscription cerulean-cloud-test-sentinel1-subscription  
   aws:lambda:Permission cerulean-cloud-test-lambda-sentinel1-permission  
@ previewing update..........................................................................................................................
~  aws:s3:BucketObject cerulean-cloud-test-titiler-lambda-archive update [diff: ~source]
~  aws:lambda:Function cerulean-cloud-test-lambda-titiler-sentinel update [diff: ~sourceCodeHash]
   aws:apigatewayv2:Integration cerulean-cloud-test-lambda-titiler-integration  
   aws:cloudwatch:MetricAlarm cerulean-cloud-test-lambda-titiler-alarm  
   aws:lambda:Permission cerulean-cloud-test-lambda-titiler-permission  
   aws:apigatewayv2:Route cerulean-cloud-test-lambda-titiler-route  
   aws:apigatewayv2:Stage cerulean-cloud-test-lambda-titiler-stage  
   pulumi:pulumi:Stack cerulean-cloud-test running Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   pulumi:pulumi:Stack cerulean-cloud-test running Checking Docker is available...
   pulumi:pulumi:Stack cerulean-cloud-test running Building container image...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully built container image with id sha256:852cce531e3bf5df86bfaa9d44dec9431843db4c5fcf2a515dcac672af1dec67
   pulumi:pulumi:Stack cerulean-cloud-test running Creating installation package.zip ...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip
   pulumi:pulumi:Stack cerulean-cloud-test  9 warnings; 6 messages
Diagnostics:
 pulumi:pulumi:Stack (cerulean-cloud-test):
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.

   Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   Checking Docker is available...
   Building container image...
   Sucessfully built container image with id sha256:852cce531e3bf5df86bfaa9d44dec9431843db4c5fcf2a515dcac672af1dec67
   Creating installation package.zip ...
   Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip

Resources:
   ~ 9 to update
   +-3 to replace
   12 changes. 55 unchanged

alukach added 2 commits December 16, 2024 14:34

Blank out metadata to make zip builds deterministic

d63fc79

Rework for better deterministic writes

8980e42

alukach temporarily deployed to test December 17, 2024 21:41 — with GitHub Actions Inactive

alukach temporarily deployed to test December 17, 2024 23:30 — with GitHub Actions Inactive

Make filename stable across deployments

806147d

alukach force-pushed the refactor/deterministic-builds branch from ec575c6 to 806147d Compare December 18, 2024 23:17

alukach temporarily deployed to test December 18, 2024 23:26 — with GitHub Actions Inactive

alukach force-pushed the refactor/deterministic-builds branch from ca4c10d to 806147d Compare December 20, 2024 18:13

alukach temporarily deployed to test December 20, 2024 18:21 — with GitHub Actions Inactive

alukach had a problem deploying to test December 20, 2024 18:22 — with GitHub Actions Failure

alukach temporarily deployed to test December 20, 2024 18:40 — with GitHub Actions Inactive

Experiment: Cache pulumi plugins

c01bfc2

alukach temporarily deployed to test December 20, 2024 19:09 — with GitHub Actions Inactive

Experiment: use GHA cache for docker builds

2f68623

alukach force-pushed the refactor/deterministic-builds branch from b5662bd to 296f13d Compare December 20, 2024 19:54

Experiment: use GHA cache for docker builds [fix]

d8507a3

alukach force-pushed the refactor/deterministic-builds branch from 296f13d to d8507a3 Compare December 20, 2024 20:02

alukach temporarily deployed to test December 23, 2024 19:43 — with GitHub Actions Inactive

Experiment: Test caching from last built image

b476f91

alukach force-pushed the refactor/deterministic-builds branch from 513914e to b476f91 Compare December 23, 2024 22:19

alukach temporarily deployed to test December 23, 2024 22:30 — with GitHub Actions Inactive

alukach temporarily deployed to test December 23, 2024 23:25 — with GitHub Actions Inactive

alukach temporarily deployed to test December 24, 2024 00:10 — with GitHub Actions Inactive

Set deployment concurrency

aa94939

alukach changed the title ~~Refactor: Deterministic builds~~ IN PROGRESS: Deterministic builds Dec 24, 2024

alukach closed this Dec 24, 2024

alukach deleted the refactor/deterministic-builds branch December 24, 2024 05:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IN PROGRESS: Deterministic builds #127

IN PROGRESS: Deterministic builds #127

alukach commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

IN PROGRESS: Deterministic builds #127

IN PROGRESS: Deterministic builds #127

Conversation

alukach commented Dec 16, 2024 • edited Loading

Before Change

After Change

github-actions bot commented Dec 16, 2024 • edited Loading

🍹 preview on cerulean-cloud-images/test

github-actions bot commented Dec 16, 2024 • edited Loading

🍹 preview on cerulean-cloud/test

alukach commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

🍹 `preview` on cerulean-cloud-images/test

github-actions bot commented Dec 16, 2024 •

edited

Loading

🍹 `preview` on cerulean-cloud/test